Alternative protocol over physical layer

ABSTRACT

A link controller includes a Peripheral Component Interconnect Express (PCIe) physical layer circuit for coupling to a communication link and providing a data path over the communication link, a first data link layer controller which operates according to a PCIe protocol, and a second data link layer controller which operates according to a Gen-Z protocol. A multiplexer-demultiplexer selectively connects both data link layer controllers to the PCIe physical layer circuit. A protocol translation circuit is coupled between the multiplexer-demultiplexer and the second data link layer controller, the protocol translation circuit receiving traffic data from the second data link layer controller in a Gen-Z format, encapsulating the Gen-Z format in a PCIe format, and passing traffic data to the multiplexer-demultiplexer circuit.

BACKGROUND

System interconnect bus standards provide for communication betweendifferent elements on a chip, or different elements with a multi-chipmodule, a circuit board, a server node, or in some cases an entireserver rack or a networked system. For example, the popular PeripheralComponent Interconnect Express (PCIe or PCI Express) is a high-speedserial expansion bus providing interconnection between elements on amotherboard, and connecting to expansion cards. Improved systeminterconnect standards are needed for multi-processor systems, andespecially systems in which multiple processors on different chipsinterconnect and share memory.

The interconnection of multi-processor computing resources and theassociated memory poses several challenges. Generally, memory capacityrequirements increase as the number of interconnected processors andaccelerators increases. Furthermore, new interconnect standards may beincompatible with older standards, such as PCIe, and therefore renderobsolete various system components and expansion devices which employthe older standards.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates in block diagram form a data processing platform witha PCIe memory module according to the prior art.

FIG. 2 illustrates in block diagram form a data processing platformaccording to some embodiments.

FIG. 3 illustrates in block diagram form another data processingplatform according to some embodiments.

FIG. 4 shows in flow diagram form a state diagram for operating a priorart link training and status state machine (LTSSM).

FIG. 5 shows in flow diagram form an example process for selecting analternative protocol using an enhanced LTSSM according to someembodiments.

FIG. 6 shows in symbol sequence diagram form an unmodified orderedtraining set according to some embodiments.

FIG. 7 shows in symbol sequence diagram from a modified ordered trainingset according to some embodiments.

FIG. 8 illustrates in block diagram form a data processing platformaccording to some embodiments.

FIG. 9 shows in flow diagram form a process for operating a dualprotocol link controller like that of FIG. 8 according to someembodiments.

FIG. 10 shows in flow diagram form another process for operating a dualprotocol link controller according to some embodiments.

FIG. 11 shows in diagram form a sequence of transmitted data on a PCIedata link according to some embodiments.

FIG. 12 shows in diagram form another sequence of transmitted data onanother PCIe data link according to some embodiments.

In the following description, the use of the same reference numerals indifferent drawings indicates similar or identical items. Unlessotherwise noted, the word “coupled” and its associated verb formsinclude both direct connection and indirect electrical connection bymeans known in the art, and unless otherwise noted any description ofdirect connection implies alternate embodiments using suitable forms ofindirect electrical connection as well.

DETAILED DESCRIPTION OF ILLUSTRATIVE EMBODIMENTS

A link controller includes a Peripheral Component Interconnect Express(PCIe) physical layer circuit, first and second data link layercontrollers, a multiplexer-demultiplexer, and a protocol translationcircuit. The link controller is connected to a communication link andprovides a data path over the communication link. The first data linklayer controller operates according to a PCIe protocol, and the seconddata link layer controller operates according to a Gen-Z protocol. Themultiplexer-demultiplexer is coupled to the first data link layercontroller, the second data link layer controller, and the physicallayer circuit. The protocol translation circuit is coupled between themultiplexer-demultiplexer and the second data link layer controller, andreceives traffic data from the second data link layer controller in aGen-Z format, encapsulates the Gen-Z format in a PCIe format, and passestraffic data to the multiplexer-demultiplexer circuit.

A method includes selectively causing a multiplexer-demultiplexer tocouple a PCIe physical layer circuit through a protocol translationcircuit to a Gen-Z data link layer controller. At the protocoltranslation circuit, traffic data is received from the Gen-Z data linklayer controller in a Gen-Z format. The Gen-Z formatted traffic data isencapsulated in a PCIe format, and passed through themultiplexer-demultiplexer to the PCIe physical layer circuit. Thetraffic data is transmitted with the encapsulated format over acommunication link from the PCIe physical layer circuit.

A data processing platform includes a central processing unit and adual-protocol link controller connected to the central processing unit.The dual-protocol link controller includes a PCIe physical layercircuit, first and second data link layer controllers, amultiplexer-demultiplexer, and a protocol translation circuit. The PCIephysical layer circuit is coupled to a communication link and provides adata path over the communication link. The first data link layercontroller operates according to a PCIe protocol, and the second datalink layer controller operates according to a Gen-Z protocol. Themultiplexer-demultiplexer is coupled to the first data link layercontroller, the second data link layer controller, and the physicallayer circuit. The protocol translation circuit is coupled between themultiplexer-demultiplexer and the second data link layer controller, andreceives traffic data from the second data link layer controller in aGen-Z format, encapsulates the Gen-Z format in a PCIe format, and passestraffic data to the multiplexer-demultiplexer circuit.

FIG. 1 illustrates in block diagram form a data processing platform 100with a PCIe memory module 120 according to the prior art. Dataprocessing platform 100 includes a processor 110 having a memorycontroller 112 and a PCIe port 114 connected to a PCIe bus 150.Expansion memory for data processing platform 100 is provided by a PCIememory module 120 connected to PCIe bus 150. PCIe memory module 120includes a memory controller 122 communicating with PCIe bus 150, and astorage class memory (SCM) 124, including multiple memory chipsproviding persistent memory storage.

FIG. 2 illustrates in block diagram form a data processing platform 200according to some embodiments. Processor 210 communicates with a memorymodule 230 using a Gen-Z protocol, which is a data access technology forenhancing memory solutions for existing and emerging memorytechnologies. The Gen-Z protocol is found in Gen-Z Core Specification1.0 published by Gen-Z Consortium, Inc., and in later versions of thestandard. Gen-Z provides an abstract device interface that supports avariety of memory types, including a number of byte-addressable,persistent storage class memory technologies. Gen-Z provides a platformfor fabric attached memory, scaling from point-to-point connections tolocal memory expansion through local high-speed busses and switchedbusses, to rack-scale solutions. To support a variety of current andfuture memory subsystems, Gen-Z provides a universal interface betweenprocessors and their memory subsystems. With this interface, thecomponents communicate using memory-semantic requests withapplication-specific semantic overlays to derive meaning and drivetype-specific actions. Generally, to communicate via Gen-Z protocol,processors and memory modules would interact using a Gen-Z PhysicalLayer, which is significantly different than a PCIe Physical Layer.Communicating using a PCIe Physical Layer provides advantages in theefficiency of PHY designs and component pinouts, and provides atransition path for adoption of Gen-Z technology. In a PCIeprotocol-based system, processors communicate with memory modules usingPCIe protocol over a PCIe Physical Layer. Host processor 210communicates with memory module 230 over a PCIe bus 220, but is able torecognize that a Gen-Z device is connected and configure a dual-protocollink controller 209 to communicate using a Gen-Z protocol as analternative protocol over a PCIe Physical Layer.

Host processor 210 includes four processor cores 202, interconnected byan on-chip interconnect network 204. This number of processor cores 202is merely an example, and the processor cores employed for various dataprocessing platforms will often include many more processor cores, suchas 32 or 64 cores all connected with an on-chip interconnect network. Asshown, on-chip interconnect network 204 links each processor core to aPCIe input of dual-protocol link controller 209 for PCIe traffic, and toGen-Z management block 206 for memory access to memory module 230. Inthis embodiment, dual-protocol link controller 209 includes a PCIeexternal port including PCIe hardware enhanced to include Gen-Zalternative protocol capability. This capability is provided by Gen-Ztransaction layer controller 211, Gen-Z data link layer controller 213,and the PCIe physical layer circuit 216. Dual-protocol link controller209 provides a Gen-Z protocol interconnect to memory module 230 that isoverlaid on a PCIe physical link on PCIe bus 220.

Gen-Z management block 206 typically includes processor memorymanagement logic, and may include other logic circuits such as requestqueues or a memory directory. On-chip interconnect network 204 sends andreceives memory requests and responses through a Gen-Z management block206, which prepares and formats messages according to the Gen-Zprotocol. Gen-Z management block 206 connects to Gen-Z transaction layercontroller 211 of dual-protocol link controller 209.

Dual-protocol link controller 209 includes a Gen-Z transaction layercontroller 211 which is connected to Gen-Z management block 206 forcommunicating memory access requests through Gen-Z management block 206in the upstream direction. Gen-Z transaction layer controller 211 isconnected to Gen-Z data link layer controller 213 for providing andreceiving Gen-Z packets in the downstream direction. Gen-Z data linklayer controller 213 generally manages the Gen-Z communication linkthrough PCIe bus 220, performing link setup, sequencing packets, andcontrolling the flow of data over the link.

Multiplexer-demultiplexer 215 selectively connects PCIe physical layercircuit 216 to Gen-Z data link layer controller 213 or PCIe data linklayer controller 214, allowing a Gen-Z link or a PCIe link to becompleted through PCIe physical layer circuit 216. PCIe physical layercircuit 216 is connected to multiplexer-demultiplexer 215, and operatesto create signals for transmission over PCIe bus 220 through aunidirectional transmit port labeled “TX”, and to receive signals overunidirectional receive port labeled “RX”. The operation ofmultiplexer-demultiplexer 215 is controlled by settings provided duringinitialization of dual-protocol link controller 209 by link training andstatus state machine (LTSSM) 217, as further described below.

On-chip interconnect network 204 includes another path for processors202 to communicate through dual-protocol link controller 209 using thePCIe protocol through connection to PCIe transaction layer controller212. This path is provided for normal PCIe traffic, allowing PCIecapable devices to be connected to PCIe bus 220 as an alternative or inaddition to the memory module 230, which operates with a Gen-Z protocol.A PCIe device may be connected to PCIe bus 220 instead of memory module230. PCIe transaction layer controller 212 is connected to PCIe datalink layer controller 214, which selectively connects to PCIe physicallayer circuit 216 through multiplexer-demultiplexer 215, as furtherdescribed below. PCIe transaction layer controller 212 and PCIe DataLink layer controller 214 operate as known in the art.

The blocks of dual-protocol link controller 209 can be implemented withvarious combinations of hardware, firmware, and software. In thisembodiment, dual-protocol link controller 209 is implemented entirely inhardware. In another exemplary implementation, PCIe physical layercircuit 216 is implemented in hardware, PCIe transaction layercontroller 212 is implemented in software, and PCIe data link layercontroller 214 is implemented partially in hardware and partially insoftware. Gen-Z management block 206 is implemented in software, Gen-Ztransaction layer controller 211 is implemented partially in hardwareand partially in software, and Gen-Z data link layer controller 213 isimplemented in hardware.

Memory module 230 may be an expansion card type module with a PCIeconnector, or may take the form of other expansion modules and/or bebuilt into the motherboard carrying host processor 210. Memory module230 includes a memory 234 with one or more memory chips connected to aninterface controller 231 over a high-speed local bus. Interfacecontroller 231 includes a media controller 232, a Gen-Z management block206, and a link controller 233. Media controller generally executesmemory access requests to memory 234. Gen-Z management block 206 isconnected to media controller 232 and prepares and formats messagesaccording to the Gen-Z protocol. Gen-Z management block 206 serves as alogical port for Gen-Z communications from media controller 232, andconnects to Gen-Z transaction layer controller 211 of link controller233.

Link controller 233 includes Gen-Z transaction layer controller 211,Gen-Z data link layer controller 213, PCIe physical layer circuit 216,and LTSSM 217, which operate similarly to those elements indual-protocol link controller 209. However, in link controller 233, noPCIe transaction layer, data link layer, or multiplexer are employed,allowing link controller 233 to only communicate with the Gen-Zprotocol. PCIe physical layer circuit 216 of link controller 233 isconnected to the transmission medium of PCIe bus 220 and transmits andreceives Gen-Z protocol communications over PCIe bus 220. Multiplechannels or a single channel may be used in the connection, running overmultiple lanes of PCIe bus 220. LTSSM 217 of link controller 233performs the functions of a PCIe LTSSM and negotiates use of the Gen-Zprotocol as described below.

Memory module 230 may be used in a memory-centric architecture or atraditional, processor-centric architecture as each is supported byGen-Z. In this example, memory 234 is a storage class memory (SCM) andis a nonvolatile memory (NVM). However, these examples are not limiting,and many types of memory modules may employ the techniques describedherein. For example, a RAM memory, or a memory with mixed NVM and RAMmay be used, such as a high-capacity flash storage or 3D crosspointmemory with a RAM buffer.

Media controller 232 may be integrated on an interface controller chip(231) with some or all of the port circuitry of dual-protocol linkcontroller 209. The two LTSSMs 217 negotiate with each other during linkinitialization to notify host processor 210 that a Gen-Z device ispresent on PCIe bus 220, and to negotiate the connection protocolbetween host processor 210 and memory module 230. This negotiationpreferably takes place as an addition to LTSSM training process that ispart of PCIe link controllers as further described below.

FIG. 3 illustrates in block diagram form a data processing platform 300.Generally, host processor 310 connects to a memory module 330 over aPCIe bus 320, recognizes that a Gen-Z capable device is connected, andconfigures the dual-protocol link controllers 309 of host processor 310and memory module 330 accordingly. Host processor 310 is identical tohost processor 210 of FIG. 2, with reference numbers to thecorresponding elements beginning with “3” rather than “2”.

Memory module 330 may be an expansion card type module with a PCIeconnector, or may take the form of other expansion modules and or bebuilt into the motherboard carrying host processor 310. Memory module330 includes a memory 334 with one or more memory chips, and aninterface controller 331. Interface controller 331 includes a mediacontroller 332 and a dual-protocol link controller 309 connected to thetransmission medium of PCIe bus 320. Multiple channels or a singlechannel may be used in the connection, running over multiple lanes ofPCIe bus 320.

Media controller 332 and its associated Gen-Z management block 306operate to fulfill and respond to memory requests formatted in thememory semantic form provided by the Gen-Z protocol. Memory module 330may be used in a memory-centric architecture or a traditional,processor-centric architecture as each is supported by Gen-Z. In thisexample, memory 334 is a storage-class, nonvolatile memory similar tothat of memory module 230.

Media controller 332 may be integrated on a chip with interfacecontroller 331, with some or all of the circuitry of dual-protocol linkcontroller 309. Dual-protocol link controller 309 has elements 311, 313,315, 316, 317, 312 and 314 like those of dual-protocol link controller309 of host processor 310, except that host processor 310 may include afull PCIe root complex in its dual-protocol link controller 309. The twoLTSSMs 317 negotiate with each other during link initialization tonotify host processor 310 that a Gen-Z capable device is present on PCIebus 320, and to negotiate the connection protocol between host processor310 and memory module 330 as further described below with respect toFIG. 5. Dual-protocol link controller 309 may be configured, typicallythrough a register or pin-strap setting, to negotiate use of either aGen-Z protocol or a PCIe protocol. This negotiation preferably takesplace as an addition to LTSSM training process that is part of PCIe linkcontrollers.

FIG. 4 shows in flow diagram form a state diagram 400 for operating aprior art PCIe LTSSM. As set forth the PCIe standard, the LTSSMtypically provides a physical layer control process that configures andinitializes each link for operation. The LTSSM performs the functions ofconfiguring and initializing the PCIe link, supporting packet transfers,recovering from link errors, and restarting a PCIe port from low powerstates. While configuring and initializing the PCIe link, the LTSSMfirst enters a Detect state where it detects the presence of a linkpartner on the lane, typically in response to the physical layer circuitbeing initialized, or commanded by the link layer as shown. From theDetect state, the LTSSM enters a Polling state in which bit and symbollock, and lane polarity are established as the link partners exchangepredetermined ordered sets of symbols, referred to as training set 1,“TS1” and training set 2, “TS2”. These ordered sets contain patterns ofbits that allow the transmitter and receiver to measure and adjust thetransmitter and receiver performance over the particular transmissionmedium of each lane.

Then the LTSSM goes to the Configuration state, where the TS1 and TS2ordered sets are again exchanged and parameters such as data rate, laneordering, and link width are established. Then the LTSSM goes to the L0,which is the normal working state where data is transferred on the link.Various errors in the Configuration process can cause the LTSSM to go tothrough the Recovery state. The LTSSM may also go to electrical idle orstandby state (L0s), lower power standby/slumber state (L1), a low powersleep state (L2), or a link Off state (L3).

FIG. 5 shows in flow diagram form an example process 500 for selectingan alternative protocol using an enhanced LTSSM 317 according to someembodiments. Generally, process 500 is conducted by the enhanced LTSSM317 controlling the PCIe physical layer circuit 316 (FIG. 3) at each endof a lane of the PCIe bus 320. Process 500 begins at process block 502where enhanced LTSSM 317 is started, typically when the data processingplatform is powered on or reset according to any suitable process, forexample a cold reset or a hot reset. The enhanced LTSSM 317 may also berestarted in response to a command from the host processor, such as acommand to leave a link standby state. At block 504, the enhanced LTSSM317 completes the Detect state to detect the presence of a physicallayer circuit transmitter or receiver at the opposite end of the lane.Then at block 506, enhanced LTSSM 317 checks for alternative protocolsenabled on the attached device. The setting is typically initializedusing a boot strap ROM, a pin set to a particular value, or some othersuitable technique to set a designated value in the Gen-Z deviceregister. Then the Gen-Z device checks the register to see that analternative protocol, such as a Gen-Z transaction layer protocol, isenabled. The check is performed at the Gen-Z device end of the link todetermine the preferred protocol with which to communicate, and may alsobe performed independently at the host processor end to determinewhether an alternative protocol is supported or allowed. At the hostprocessor, the setting may be stored in a register, which is checked bythe PCIe root complex.

If no alternative protocols are enabled, process 500 at block 508 goesto the normal PCIe LTSSM process where it completes the polling state atblock 516, the configuration state at block 518, and upon completing thelink configuration at block 520, exits to the LO operating state atblock 522. Block 518 may configure multiplexer-demultiplexer 315 of bothdual-protocol link controllers 309 at either end of the link to connectthe PCIe data link layer controller 314 to PCIe physical layer circuit316, or such connection may already be set as the default state. If thePCIe protocol is not enabled as a default, block 518 may also includetransmitting PCIe protocol negotiation information identifying the PCIeprotocol in the same manner that alternative protocol negotiationinformation is exchanged at block 510.

If an alternative protocol is enabled, referring to block 508, process500 goes to block 510, where it negotiates the use of an alternativeprotocol by transmitting modified TS1 and TS2 ordered sets. The orderedsets are modified to insert information into the TS1 or TS2 set at theGen-Z device end of the link indicating an alternative protocol issupported. The enhanced LTSSM 317 transmits and receives alternativeprotocol negotiation information over the data lane within the modifiedTS1 and TS2 ordered sets. The host processor 310 end of the linksimilarly acknowledges acceptance of the alternative protocol byinserting acknowledgement information into the TS1 or TS2 ordered settransmitted back to memory module 330.

At block 512, process 500 configures the multiplexer-demultiplexer 315of both dual-protocol link controllers 309 at either end of the link, toconnect the Gen-Z data link layer controller to PCIe physical layercircuit 316. Generally, if a Gen-Z protocol is supported at both ends ofthe link, the host processor 310 and memory module 330, the Gen-Z datalink layer controller is used. If either end of the link only supportsthe PCIe protocol, then the PCIe data link layer controller 314 is used.At block 514, the configuration of the lane is completed by negotiatinglink speed, link width, and other relevant parameters.

This scheme enables the use of PCIe or Gen-Z communication in a mannertransparent to the application layers of the system. It also allows thesame physical transmission medium, the lanes of PCIe bus 320, often 16lanes, to be used by both protocols. Because the alternative protocolnegotiation is done on a lane-by-lane basis, a number of lanes may beused for a Gen-Z protocol (for example, a memory module) while otherlanes are used for a PCIe protocol (for example, for peripheraldevices). The techniques herein also allow for backward compatibility,as older PCIe devices do not interfere with the Gen-Z specific hardware.Further, use of these techniques within a data fabric allows processingelements multiple paths to reach a port of their choice and select aprotocol of their choice.

FIG. 6 shows in symbol sequence diagram form an unmodified orderedtraining set 602 according to some embodiments. The unmodified trainingset generally consists of two types of symbol sets, each having 16symbols used by the LTSSM to establish alignment and other linkparameters during the Polling and Configuration states of the LTSSM.

FIG. 7 shows in symbol sequence diagram from a modified ordered trainingset 702 according to some embodiments. Modified ordered training set 702may be a modified version of either or both of the TS1 or TS2 orderedtraining sets used by the LTSSM. Modified data 704 includes alternativelink negotiation parameters, which identify the protocol to be employed,such as a Gen-Z protocol. The modified data includes at least one bitaltered from the original TS1 or TS2 ordered set. The enhanced LTSSM 317checks the position of the modified bits for altered data to determineif a modified ordered training set is received.

FIG. 8 illustrates in block diagram form a data processing platform 800according to some embodiments. Generally, a host processor 810 connectsto a memory module 830 over a PCIe bus 820, recognizes that a Gen-Zdevice is connected, and configures the dual protocol link controllers809 of host processor 810 and dual protocol link controller 833 ofmemory module 830 accordingly. In this embodiment, a protocoltranslation circuit 818 is present in dual-protocol link controllers 809and 833, which operates to convert or encapsulate data from the Gen-Zdata link layer controller 813 with a format suitable for transmissionwith PCIe physical layer circuit, as further described below.

Host processor 810 has many similar elements to host processor 210 ofFIG. 2, with reference numbers to the corresponding elements beginningwith “8” rather than “2”. The description of these elements will not berepeated unless pertinent to the present embodiment.

In this embodiment, dual protocol link controller 809 includes aGen-Z/PCIe external port including PCIe hardware enhanced to includeGen-Z alternative protocol capability. This capability is provided bythrough Gen-Z management block 206, Gen-Z transaction layer controller811, Gen-Z data link layer controller 813, protocol translation circuit818, multiplexer-demultiplexer 815, and PCIe physical layer circuit 816.Dual protocol link controller 809 provides a Gen-Z protocol interconnectto memory module 830 that is overlaid on a PCIe physical link on PCIebus 820.

Memory module 830 may be an expansion card type module with a PCIeconnector, or may take the form of other expansion modules and or bebuilt into the motherboard carrying host processor 810. Memory module830 includes a memory 834 with one or more memory chips, and aninterface controller 831. Interface controller 831 includes a mediacontroller 832 and a dual protocol link controller 833 connected to thetransmission medium of PCIe bus 820. Multiple channels or a singlechannel may be used in the connection, running over multiple lanes ofPCIe bus 820. Media controller 832 may be integrated with some or all ofthe port circuitry of dual protocol link controller 833 of interfacecontroller 831 on a single chip.

Media controller 832 and its associated Gen-Z protocol layer 806 operateto fulfill and respond to memory requests formatted in the memorysemantic form provided by the Gen-Z protocol. Memory module 830 may beused in a memory-centric architecture or a traditional,processor-centric architecture as each is supported by Gen-Z. In thisexample, memory 834 is a storage-class, nonvolatile memory.

Dual-protocol link controller 833 has elements 811, 813, 816, similar tothose described above with respect to FIG. 2. In this embodiment, dualprotocol link controller 833 includes a protocol negotiation circuit817, which negotiates with a similar protocol negotiation circuit 817 ofdual protocol link controller 809. Protocol negotiation circuits 817negotiate with each other during link initialization to notify hostprocessor 810 that a Gen-Z device is present on PCIe bus 820, and tonegotiate the connection protocol between host processor 810 and memorymodule 830. Negotiation may occur through a LTSSM as described withrespect to FIG. 5, for example, or any other suitable method. Generally,protocol negotiation information is exchanged during the link setup. Forexample, a single bit in an ordered set exchanged during the PCIe linksetup, such as a Training Ordered Set, may be modified to carryGen-Z/PCIe protocol negotiation information.

A protocol translation circuit 818 is included in dual protocol linkcontroller 809 and dual protocol link controller 833. In both instances,protocol translation circuit 818 is connected to Gen-Z data link layercontroller 813. In the direction toward the PCIe bus 820, protocoltranslation circuit 818 connects to multiplexer-demultiplexer 815 indual protocol link controller 809. In memory module 830, protocoltranslation circuit 818 connects to PCIe physical layer circuit 816.Protocol translation circuits 818 are preferably constructed as digitalcontrollers including communication buffers.

Protocol translation circuit 818 of dual protocol link controller 809generally operates to encapsulate Gen-Z formatted traffic data in a PCIeformat, and pass the traffic data to PCIe physical layer circuit 816through multiplexer-demultiplexer 815. Protocol translation circuit 818also operates to receive encapsulated traffic data from memory module830 and perform the reverse process, as further described below withrespect to FIGS. 9-13.

FIG. 9 shows in flow diagram form a process 900 for operating a dualprotocol link controller like that of FIG. 8 according to someembodiments. Referring to FIG. 8 and FIG. 9, process 900 begins at block902 with the protocol negotiation circuits 817 completing the alternateprotocol negotiation process to specify that a Gen-Z protocol will beused to communicate with an attached module such as a memory module 830.At block 904, multiplexer-demultiplexer 815 is configured to connectprotocol translation circuit 818 to PCIe physical layer circuit 816.This connection may be commanded through a control connection toprotocol negotiation circuit 817, or by a dedicated controller for dualprotocol link controller 809.

Next, at block 906, PCIe physical layer circuit 816 forms a connectionwith the Gen-Z data link layer controller 813 through protocoltranslation circuit 818. In block 906, PCIe physical layer circuit 816acts according to the PCIe protocol, asserting a digital signal toindicate a connection is available. Protocol translation circuit 818translates this signal into the signal expected by Gen-Z data link layercontroller 813. At block 908, Gen-Z data link layer controller 813responds to the initiation through protocol translation circuit 818. Theresponse typically includes asserting a digital signal to PCIe physicallayer circuit 816 in a manner expected under normal link initializationprocedures for a PCIe link. At this point the communications link isinitiated as indicated at block 910, and link controls activity such asreleasing credits may occur.

At block 912, Gen-Z data link layer controller 813 sends traffic data inpackets to protocol translation circuit 818 formatted in a Gen-Z format.This includes header information arranged as if protocol translationcircuit 818 were a Gen-Z physical layer circuit. Protocol translationcircuit 818 then encapsulates the traffic data, including the Gen-Zformatting information, into a PCIe format at blocks 914 and 916. Block914 includes dividing or grouping the Gen-Z packets into chunks sizedfor PCIe transaction layer packets (TLP). By encapsulation, it is meantthat the Gen-Z data, along with its associated framing, is exactlyreproduced at the receiver end of the data link, after being transmittedwithin the PCIe format. This provides a physical layer transmitter thatallows the Gen-Z protocol to be used with a PCIe physical layer,transparent to the Gen-Z protocol layers.

Preferably, protocol translation circuit 818 operates to format the TLPssuch that a TLP transmission does not extend beyond an interval betweena selected PCIe skip ordered set (SKP OS) and a subsequent SKP OSrequired by PCIe physical layer circuit 816. This is not to say that asingle TLP or a number of TLPs must exactly fill the gap. For example,the gap may be filled by a number of TLPs of the largest size available,plus one smaller TLP to fill the gap to where an SKP OS is expected tobe requested, followed by a number of logical idle symbols which areused until the SKP OS is actually begun. At block 916, protocoltranslation circuit 818 operates to prepare PCIe-formatted transactionlayer packets (TLPs) including data from one or more Gen-Z packets andPCIe TLP framing data or tokens. Protocol translation circuit 818operates to express the data from the one or more Gen-Z packets withPCIe symbols.

At block 918, protocol translation circuit 818 sends the prepared TLPsto PCIe physical layer circuit 816. At block 920, the TLPs aretransmitted over the communication link to memory module 830. Suchtransmission may occur over one or more PCIe lanes, depending on theconfiguration of PCIe bus 820.

It is noted that while the steps are depicted in a certain order, thisorder is not limiting and many of the operations happen simultaneouslywhen transmitting data. The depicted process exemplifies establishing anoutgoing communication link, for example from host processor 810 tomemory module 830. A similar process typically happens at dual protocollink controller 833 of memory module 830 to establish a return data linkfor two-way communication.

FIG. 10 shows in flow diagram form another process 1000 for operating adual protocol link controller according to some embodiments. Thedepicted process handles certain flow control procedures that are partof the PCIe protocol. In particular, PCIe protocols defines specialsymbols that can be inserted as SKP (skip) symbols which are only foundin SKP ordered-sets (SKP OS). For example, a transmitted SKP OS mayinclude a single COM symbol followed by a number SKP symbols. In PCIe,transmitters are required to insert SKP OS's into the transmitted datastream periodically to prevent an overflow or underflow condition in thereceiver buffer, because the clocks used at the transmitter and receivermay not be exactly matched. In some embodiments, protocol translationcircuits 818 like those of FIG. 8 handle such insertions according toprocess 1000.

Referring to FIG. 8 and FIG. 10, at block 1002, TLPs are beingtransmitted by the PCIe physical layer circuit 816. At some point duringsuch transmission, depending on clock rates and buffer states, PCIephysical layer circuit 816 asserts a request to protocol translationcircuit 818 to halt transmission for inserting an SKP OS as shown atblock 1004. Protocol translation circuit 818 receives the request andfinishes transmitting any packets that are currently being transmitted,then halts packet transmission to allow the PCIe physical layer circuitto insert a SKP OS. Protocol translation circuit 818 typically signalsback to PCIe physical layer circuit 816 that packets are not beingtransmitted, as shown at block 1006. This signaling is typically done byasserting a digital signal. Protocol translation circuit 818 may alsoassert a port enable signal for a port on which the SKP OS is requestedto be inserted, as shown at block 1008.

At block 1010, PCIe physical layer circuit 816 transmits one or more SKPOS's, inserting them into the transmitted traffic stream. The number ofSKP OS's may vary as to mitigate buffer problems on the receiver end ofthe transmission link. After the SKP OS insertion is over, thecommunication link is reestablished, and packet transmission continuesas shown at block 1012.

While this process is given as an example, protocol translation circuit818 may handle other procedures required by PCIe physical layer circuit816 that are not handled by a Gen-Z link layer controller. As such,protocol translation circuit 818 is generally adapted to receive asignal or indicator that a particular PCIe action is required foroperation of the data link over PCIe physical layer circuit 816, andprovide a response to PCIe physical layer circuit 816 to resolve therequirement. This type of process is performed by protocol translationcircuits 818 at both the transmitting and receiving ends of the datalink.

FIG. 11 shows in diagram form a sequence 1100 of transmitted data on aPCIe data link according to some embodiments. Sequence 1100 may betransmitted, for example, in a system arranged like that of FIG. 8, butwith only a single PCIe data lane used in each direction to establish adata link (an “x1” link) over PCIe bus 820. Once the PCIe physical layercircuit 816 transitions to the active state, known as the PHY-Up state,Gen-Z packets are transmitted over the data link encapsulated in PCIeTLPs in the manner shown. PCIe framing is included to follow the PCIeprotocol and encapsulate the Gen-Z traffic data.

In sequence 1100, TLPs labeled as TLP 0-TLP P are transmitted, separatedwith a synch header, shown as “Synch Hdr.” Each TLP starts and at leastone STP token, or in this case four STP tokens as can be seen fillingthe first four transmitted symbols of TLP 0 and TLP 1. The TLPs mayencapsulate one or more consecutive Gen-Z packets, including partialGen-Z packets. Gen-Z packets may span TLP boundaries, as depicted byGen-Z packet N, which begins at symbol 15 of TLP 0 and is continued andcompleted in TLP 1 symbols 4 through 10.

Generally the structure of the TLPs follows PCIe standards. TLP lengthis within the minimum and maximum TLP length requirements as specifiedby the PCI Express Base Specification. PCIe Ordered-Sets and PCIeFraming Tokens, as used in the PCIe standards, are removed from receiveddata before transmitting the received data to Gen-Z data link layercontroller 813. Additional format restrictions may also be applied thatconform to PCIe standards but place further rules on the format. Asdescribed with respect to FIG. 10, SKP OS's may be inserted in the datastream between TLPs. Where such insertion is required, Gen-Z packettransmission is halted to insert SKP Ordered-Sets as specified by thePCIe Base Specification. Gen-Z packet transmission halts when the datastream needs to transmit SKP ordered-sets or other ordered setsassociated with PHY link management or link state transitions, asdepicted at the end of sequence 1100.

Sequence 1100 uses 128b/130b PCIe encoding. Generally, PCIe physicallayer circuit 816 supports PCIe data framing using the 8b/10b or128b/130b PCIe encoding rules, as required. The requirements of PCIethat relate to link layer functionality are not followed, because theyare subsumed or rendered irrelevant by the use of Gen-Z data link layercontroller 813. The LCRC (link cyclic redundancy check), which isnormally generated by the transmitter or checked at the receiver, is notused. Instead, Gen-Z packet data occupies the location where LCRC isnormally located. Also, TLP sequence numbers are always 0, and PCIepacket retry is not used. Instead, Gen-Z retry mechanisms are used fortransient error recovery, and Gen-Z ECRC (end-to-end CRC), PCRC (preludeCRC) and optional flit CRC are used for Gen-Z packet protection andpacket boundary determination. All framing rules specified in the PCIExpress Base Specification regarding STP, IDL (idle symbol), EDS (enddata stream symbol), and Ordered-Sets are followed.

The TLPs in sequence 1100 start with an STP token with the TLP length,ECRC, and parity and a TLP sequence number of 0. The STP tokens followedby scrambled and 128b/130b encoded data blocks containing Gen-Z packetsfrom the Gen-Z Core. Received 128b/130b data symbols are decoded anddescrambled from TLPs as specified by the PCI Express Base Specificationto un-encapsulate the Gen-Z packets.

FIG. 12 shows in diagram form another sequence 1200 of transmitted dataon another PCIe data link according to some embodiments. Sequence 1200is an example of data transmitted using four PCIe data lanes (an “x4”link) to transmit data over PCIe bus 820, with the lanes depicted acrossthe table horizontally and the symbols transmitted in each timeslotshown vertically down the page.

As shown in sequence 1200, the start of a TLP with a STP symbol followedby a Gen-Z packet stream. TLP 0 includes Gen-Z packets 0-M, and part ofpacket N, with the remainder of packet N transmitted in TLP 1. Sincethis example sequence is 8b/10b encoded, synch headers are not used. Theencapsulated Gen-Z packet data scrambled and encoded as data symbolsusing 8b/10b scrambling and encoding rules for TLP packets as specifiedby the PCI Express Base Specification. At the end of the TLP, the fourbytes of LCRC typically required by PCIe are replaced by four bytes of aGen-Z packet stream, and followed by an END symbol to end the TLP.Received 8b/10b data symbols are decoded and descrambled as specified bythe PCI Express Base Specification to de-encapsulate Gen-Z packets.

In this example, the buffer conditions of the receiver require that SKPordered sets be transmitted over the data link. As can be seen aftersymbol 1535, transmission of Gen-Z packets is halted and a sequence ofCOM and SKP symbols are transmitted by the PCIe physical layer circuit.Then the transmission resumes with TLP 1, picking up partially throughGen-Z packet N and continuing transmission.

While the encoding used is 8b/10b encoding for this example, 128b/130bencoding may also be used with multiple data lanes. Future physicallayer encoding schemes may also be used. Generally, all supported datawidths in PCIe may be used with the techniques herein, including thosewidths achieved by data striping.

The techniques herein may be used, in various embodiments, with anysuitable products (e.g. servers, data processing computers, databasehosts) that employ memory modules or other peripherals that benefit fromhigh speed communication links. Further, the techniques are broadlyapplicable for use in data processors implemented with GPU and CPUarchitectures or ASIC architectures, as well as programmable logicarchitectures.

While particular embodiments have been described, various modificationsto these embodiments will be apparent to those skilled in the art. Forexample, various kinds of modules may benefit from using the linkcontroller designs herein.

The systems and circuits depicted in FIG. 2, FIG. 3, and FIG. 8 may bedescribed or represented by a computer accessible data structure in theform of a database or other data structure which can be read by aprogram and used, directly or indirectly, to fabricate integratedcircuits. For example, this data structure may be a behavioral-leveldescription or register-transfer level (RTL) description of the hardwarefunctionality in a high level design language (HDL) such as Verilog orVHDL. The description may be read by a synthesis tool which maysynthesize the description to produce a netlist including a list ofgates from a synthesis library. The netlist includes a set of gates thatalso represent the functionality of the hardware comprising integratedcircuits. The netlist may then be placed and routed to produce a dataset describing geometric shapes to be applied to masks. The masks maythen be used in various semiconductor fabrication steps to produce theintegrated circuits. Alternatively, the database on the computeraccessible storage medium may be the netlist (with or without thesynthesis library) or the data set, as desired, or Graphic Data System(GDS) II data.

Accordingly, it is intended by the appended claims to cover allmodifications of the disclosed embodiments that fall within the scope ofthe disclosed embodiments.

1. A link controller comprising: a Peripheral Component InterconnectExpress (PCIe) physical layer circuit coupled to a communication linkand providing a data path over said communication link; a first datalink layer controller which operates according to a PCIe protocol; asecond data link layer controller which operates according to a Gen-Zprotocol; a multiplexer-demultiplexer coupled to the first data linklayer controller and the PCIe physical layer circuit; and a protocoltranslation circuit coupled to the multiplexer-demultiplexer and thesecond data link layer controller, the protocol translation circuitreceiving traffic data from the second data link layer controller in aGen-Z format, encapsulating the Gen-Z format traffic data in a PCIeformat, and passing the encapsulated traffic data to themultiplexer-demultiplexer.
 2. The link controller of claim 1, whereinthe protocol translation circuit receives an indication from the PCIephysical layer circuit that the communication link is ready to initiatea connection, transmits an indication to the PCIe physical layer circuitto initiate a connection, and during the connection passes only datafrom the second data link layer controller through themultiplexer-demultiplexer to the PCIe physical layer circuit.
 3. Thelink controller of claim 1, wherein the protocol translation circuitoperates to prepare PCIe-formatted transaction layer packets (TLPs)including data from one or more Gen-Z packets and PCIe framing data. 4.The link controller of claim 3, wherein the protocol translation circuitoperates to format the TLPs such that a TLP transmission does not extendbeyond an interval between a selected PCIe skip ordered set (SKP OS) anda subsequent SKP OS required by the PCIe physical layer circuit.
 5. Thelink controller of claim 3, wherein the protocol translation circuitoperates to express the traffic data from the one or more Gen-Z packetswith PCIe symbols.
 6. The link controller of claim 1, wherein theprotocol translation circuit receives a request from the PCIe physicallayer circuit to provide a gap in data transmission for inserting a PCIeskip ordered set (SKP OS), and in response, halts packet transmission toallow the PCIe physical layer circuit to insert a SKP OS.
 7. The linkcontroller of claim 1 further comprising a protocol selection circuitwhich selectively controls the PCIe physical layer circuit to: transmitand receive alternative protocol negotiation information over thecommunication link; and, in response to receiving the alternativeprotocol negotiation information, cause the multiplexer-demultiplexer toselectively couple the PCIe physical layer circuit to the second datalink layer controller.
 8. A method comprising: selectively causing amultiplexer-demultiplexer to couple a Peripheral Component InterconnectExpress (PCIe) physical layer circuit through a protocol translationcircuit to a Gen-Z data link layer controller; and at the protocoltranslation circuit, receiving traffic data from the Gen-Z data linklayer controller in a Gen-Z format, encapsulating the Gen-Z formattraffic data in a PCIe format, and passing the encapsulated traffic datathrough the multiplexer-demultiplexer to the PCIe physical layercircuit; and transmitting the encapsulated traffic data over acommunication link from the PCIe physical layer circuit.
 9. The methodof claim 8, further comprising, at the protocol translation circuit,receiving an indication from the PCIe physical layer circuit that thecommunication link is ready to initiate a connection, transmitting anindication to the PCIe physical layer circuit to initiate a connection,and during the connection passing only data from the Gen-Z data linklayer controller through the multiplexer-demultiplexer to the PCIephysical layer circuit.
 10. The method of claim 8, further comprising,at the protocol translation circuit, preparing PCIe-formattedtransaction layer packets (TLPs) including data from one or more Gen-Zpackets and PCIe framing data.
 11. The method of claim 10, furthercomprising, at the protocol translation circuit, expressing the datafrom the one or more Gen-Z packets with PCIe symbols.
 12. The method ofclaim 10, further comprising, at the protocol translation circuit,formatting the TLPs such that a TLP transmission does not extend beyondan interval between a selected PCIe skip ordered set (SKP OS) and asubsequent SKP OS required by the PCIe physical layer circuit.
 13. Themethod of claim 10, further comprising, at the protocol translationcircuit, receiving a request from the PCIe physical layer circuit toprovide a gap in data transmission for inserting a PCIe skip ordered set(SKP OS), and in response, halting packet transmission to allow the PCIephysical layer circuit to insert a SKP OS.
 14. The method of claim 8,further comprising, transmitting and receiving alternative protocolnegotiation information over the communication link; and, in response toreceiving the alternative protocol negotiation information, causing themultiplexer-demultiplexer to couple the PCIe physical layer circuitthrough a protocol translation circuit to the Gen-Z data link layercontroller.
 15. A data processing platform comprising: a centralprocessing unit; a dual-protocol link controller coupled to said centralprocessing unit and comprising: a Peripheral Component InterconnectExpress (PCIe) physical layer circuit coupled to a communication linkand providing a data path over said communication link; a first datalink layer controller which operates according to a PCIe protocol; asecond data link layer controller which operates according to a Gen-Zprotocol; a multiplexer-demultiplexer coupled to the first data linklayer controller and the PCIe physical layer circuit; and a protocoltranslation circuit to the multiplexer-demultiplexer and the second datalink layer controller, the protocol translation circuit receivingtraffic data from the second data link layer controller in a Gen-Zformat, encapsulating the Gen-Z format traffic data to a PCIe format,and passing the encapsulated traffic data to themultiplexer-demultiplexer.
 16. The data processing platform of claim 15,wherein the protocol translation circuit receives an indication from thePCIe physical layer circuit that the communication link is ready toinitiate a connection, transmits an indication to the PCIe physicallayer circuit to initiate a connection, and during the connection passesonly data from the second data link layer controller through themultiplexer-demultiplexer to the PCIe physical layer circuit.
 17. Thedata processing platform of claim 15, wherein the protocol translationcircuit operates to prepare PCIe-formatted transaction layer packets(TLPs) including data from one or more Gen-Z packets and PCIe framingdata.
 18. The data processing platform of claim 17, wherein the protocoltranslation circuit operates to format the TLPs such that a TLPtransmission does not extend beyond an interval between a selected PCIeskip ordered set (SKP OS) and a subsequent SKP OS required by the PCIephysical layer circuit.
 19. The data processing platform of claim 17,wherein the protocol translation circuit receives a request from thePCIe physical layer circuit to provide a gap in data transmission forinserting a PCIe skip ordered set (SKP OS), and in response, haltspacket transmission to allow the PCIe physical layer circuit to insert aSKP OS.
 20. The data processing platform of claim 15, furthercomprising: a memory module comprising a memory, a media controllercoupled to the memory, and an interface controller coupled to the mediacontroller and the communication link, a second PCIe physical layercircuit communicatively coupled to a second Gen-Z data link layercontroller through a second protocol translation circuit.